The present invention is directed to a network architecture for the programmable emulation of large artificial neural networks (ANN) having digital operation.
Artificial neural networks (referred to below as ANN) are proposed for the parallel processing of extremely large data sets with the goal of pattern recognition and processing (for example, of voice or images). Such an ANN is composed of a plurality of non-linear processor elements (neurons) that are networked with one another via variable "weighting factors".
The following non-linear transfer characteristics have been proposed for modeling the neurons, c.f. R. P. Lippmann, "An Introduction to Computing with Neural Nets", IEEE ASSP Magazine, April 1987, pages 4-22:
binary distinction, PA0 linear ramp function with saturation characteristic, PA0 sigmoidal function, PA0 tangent-hyperbola function. PA0 be able to synthesize neural networks of an arbitrary type and of a size adequate for the applications (modularity); PA0 allow or itself undertake the external programming of the weightings and thresholds; PA0 contain the known discriminator functions; PA0 allow the branching or non-branching of inputs or, respectively, outputs; PA0 enable the feedback of the outputs onto the (branched or non-branched) inputs; PA0 comprise the simplest interfaces to the accompanying development environment; and PA0 be employable as a module for larger emulators.
Further, a plurality of different structures for the networking of the neurons exist (for example "Hopfield net", "Hamming net", "Perceptron", c.f. R. P. Lippmann as well). A direct digital realization of large nets of this type seems impossible with the current technologies for manufacturing integrated circuits because of the plurality of neurons (&gt;1000) that are required. The main problem is that the number of connections and, thus, the number of variable weighting factors quadratically increases with the number of neurons. For example, 1,000,000 weights are required given 1000 neurons in a completely meshed network.
A few realizations of ANN having, for example, 54 or 256 neurons are disclosed in the literature for a programmable or, respectively, non-programmable network, c.f. H. P. Graf, P. De Vegvar, "A CMOS Associative Memory Chip based on Neural Networks, Proc. 1987 IEEE Int. Conf. on Solid State Circuits, pages 304, 305, 437; H. P. Graf et al, "VLSI implementation of a neural network memory with several hundreds of neurons", AIP Conference Proceedings 151, "Neural Networks for Computing", pages 182 through 187, Snowbird, Utah, 1986; W. Hubbard et al, "Electronic Neural Networks", AIP Conference Proceedings 151, "Neural Networks for Computing", pages 227 through 234, Snowbird, Utah, 1986. Proposed realizations have also been disclosed, see J. P. Sage, K. Thompson, R. S. Withers, "An Artificial Neural Network Integrated Circuit based on MNOS/CCD principles", AIP Conference Proceedings 151, "Neural Networks for Computing", pages 381 through 384, Snowbird, Utah, 1986. These all involve analog realizations of analog ANNs. Compared to digital realizations, they have the advantage of a significantly smaller implementation surface area. What is disadvantageous, however, is the high power consumption caused by the required resistance matrix, this power consumption opposing a realization of larger, programmable networks (with more than a few hundred neurons). The behavior and the properties of the modelings of ANNs hitherto proposed are preeminently investigated by simulation on vector computers, work stations or special processor fields. The fundamental disadvantage of this method is that the (space) parallelism inherent in the neural network is completely or partially lost in the processing of the information and, thus, the calculating time of the simulated network increases to such orders of magnitude, particularly for large neuron composites, that a speedy or quasi-real-time processing of the jobs cited above is impeded or becomes impossible.
A significant shortening of the calculating time and a far greater pattern throughput is obtained, by contrast, with hardware emulators. In contrast to hardware simulators, they contain an artificial neural network having a small size with whose assistance a larger network can be emulated.
In addition to a higher throughput, the advantage of a digital realization of an ANN is comprised in the greater freedom with which the user can select and set networktypical parameters (for example, discriminator functions).
Since, on the one hand, the algorithm of a neural network having no specific architecture can be executed faster, and networks with only a few hundred neurons do not present a meaningful application because of their low memory capacity, the following points must be particularly taken into consideration in the design of digital emulators for artificial neural networks (referred to below as neuroemulators): A neuro-emulator should